Color aided motion-segmentation and object tracking for video sequences semantic analysis

The high rates at which digital multimedia is being generated and used makes it necessary to develop systems that can process it in an efficient manner. This can be achieved by extracting semantics from processing the video’s low-level information. We present a novel algorithm which fuses color and motion information, in order to extract semantics from the video sequence. The motion estimates are processed statistically to give areas of activity in the video. Color segmentation is applied to these areas, and also to their complementary regions in each frame, in order to achieve the moving object segmentation. The extracted color layers in the activity and background areas are compared using the Earth Mover’s Distance (EMD), and a novel method, which we introduce, and which is based on a likelihood ratio test (LRT). The segmentation results of our LRT-based approach are shown to be more robust than the EMD results, and both methods are shown to be more accurate than existing combined color-motion approaches. Furthermore, the LRT method allows the retrieval of additional semantics, namely of “maps” that indicate with what likelihood a pixel belongs to a moving object. The areas of activity can be used to retrieve semantics for the kind of activity taking place. The color-aided segmentation of the moving entities provides a full description of their appearance, so it can be used, for example, to classify the video based on the objects in it. Experiments with real sequences show that this method leads to accurate results and useful semantics.